|
Poster:
|
JeffW |
Date:
|
October 24, 2002 12:10:32am |
|
Forum:
|
web
|
Subject:
|
crawling in 2002? |
Has the updating of the archive slowed? Most of my site haven't been rearchived at all in 2002, though the sites have definitely had many changes in 2002.
My sites (examples: wilk4.com, centralpc.org, hortico.com, brassica.com, broccosprouts.com) were archived in 2001 and previous years, but 2002 is blank.
What's going on? I don't think I've set anything in robots.txt that would stop the IA crawling. Other sites than mine that I've checked also seem to have few updates this year.
BTW, some of the faq links are broken.
|
Poster:
|
 brewster |
Date:
|
October 24, 2002 02:21:51am |
|
Forum:
|
web
|
Subject:
|
Yes we are crawling in 2002 |
Crawling is continuing by Alexa Internet (thank you to Alexa!), and the Internet Archive gets it with a 6 month time delay. But, as you observe we are behind on this. We are rebuilding indices so hopefully in the next couple of weeks we will be up to date (meaning only 6 months out of date).
-brewster
|
Poster:
|
JeffW |
Date:
|
October 24, 2002 02:35:05am |
|
Forum:
|
web
|
Subject:
|
Re: Yes we are crawling in 2002 |
good to know. You might post that info as a FAQ so people know that, at best, there is a 6 month delay.
btw, your forum login is a little funky. I login and it says it failed. If I refresh pages though it says I'm in.
TVNews
OpenLibrary